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This paper presents a transferable and adaptable approach for selecting a suitable Multi-Criteria Analysis 
(MCA) method covering the stages of the ‘MCA cycle’ for decision making in the renewable energy sector. 
Selecting a suitable MCA method for an individual project or case study is a challenge due to the 
complexity of participatory sustainable decision making in this sector, as well as due to the complexity 
and large number of existing MCA methods. The methodology presented in this paper consists of both, 
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generic and case-study specific parts. 

The paper concludes that the methodology provides a transferable approach to systematically select a 
MCA method and to justify this selection. The methodology supports focusing on specific aspects, which 
are considered important and relevant for decision making processes. Furthermore, the case study's 
generic decision making situation allows the methodology to be applied broadly within the renewable 
energy sector. Moreover, the MCA methods and selection criteria identified and reviewed provide a broad 
application potential to assist MCA method selection in the energy and renewable energy sectors. 

© 2013 Elsevier Ltd. All rights reserved. 
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1. Introduction 


criteria and indicators (C&I) [1-3]. MCA approaches show explicitly 
how the alternative options contribute to the various C&I and contain 


Multi-criteria analysis (MCA) can be defined as formal or struc- 
tured approaches for individuals or groups to determine overall 
preferences among alternative options by taking account of multiple 
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a judgment process, which usually comprises a relative weighting 
process. MCA approaches can be exercised for identifying a single 
most preferred option or for a ranking of alternative options, a short 
list with a limited number of options for further assessment, as well as 
for determining acceptable and unacceptable options [2]. They have 
become increasingly popular in decision making on sustainable 
developments and on energy systems due to their ability to consider 
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and simultaneously evaluate a number of economic, environmental, 
social and technological aspects, which complex systems require [4]. 
For that purpose MCA approaches offer algorithms to systematically 
aggregate multi-dimensional information to reduce complexities in a 
transparent way (eg. [5]). They proved to be of assistance in 
comprehensive decision processes and in overcoming implementation 
barriers by structuring the problem, identifying less robust and 
uncertain components of energy systems and involving different 
stakeholders in challenging and difficult decision making situations 
[6]. 

In literature, there are several approaches to classify MCA 
approaches. A popular form of MCA is Multi-Criteria Decision 
Analysis (MCDA), which is also known as multi attribute decision 
analysis (MADA) [2]. Further notable is that the terms ‘MCDA’, as 
well as ‘MCDM’ (Multi-Criteria Decision Making) are often syno- 
nymously used for MCA. 

Moreover, Buchholz et al. [6] and others [7-9] are describing 
two main categories: Multi Objective Decision Making (MODM) 
and Multi Attribute Decision Making (MADM) approaches. The 
main difference between the two groups of approaches is the 
number of possible alternatives, which are evaluated. For MODM 
an indefinite number of alternatives are defined by a set of 
constraints on a vector of decision variables resulting in an 
optimized alternative. On the other hand, for MADM approaches, 
which are usually employed for environmental decision making 
problems [10], a certain finite number of alternatives, which are 
given explicitly, are developed and assessed against a set of 
principles or C&I. These are often expressed as qualitative and/or 
quantitative attributes [1,7,10,11]. 

According to Belton and Stewart [1] (adopted by others, e.g. 
{6,7]) MADM approaches can generally be differentiated into three 
broad categories to illustrate their application ranges more 
directly. These three categories are: (i) value measurement mod- 
els, (ii) goal, aspiration and reference-level models and (iii) out- 
ranking models. The approaches in the first category follow the 
Multi Attribute Value Theory (MAVT) and its extension the Multi 
Attribute Utility Theory (MAUT), which works with utility func- 
tions for additional inclusion of risks and uncertainties. Belton and 
Stewart [1] describe goal, aspiration and reference-level models as 
goal programming methods, which employ algorithms based on 
mathematical programming to meet the goals, aspirations or 
reference levels as much as possible. Outranking models compare 
scenarios and alternatives against each criterion and its indicator 
on a pairwise basis [12]. In this process incomparabilities are 
identified, as well as preferences and indifferences are assessed. 
The aggregation of preferences in respect to each criterion and its 
indicator results in a scenario or alternative, which outranks 
others. However, in contrast to MAUT the aim is usually not to 
identify a single highest scoring alternative, but rather to involve 
multiple stakeholders in a process to advise each other | 13]. 

MCA methods can be considered for different stages of multi- 
criteria decision making. Wang et al. [14] reviewed MCDA methods 
for C&I selection, C&I weighting, evaluation, and final aggregation. 
However, with respect to the latter stage Guitouni and Martel [15] 
believe that aggregation procedures and MCA methods are con- 
fusing concepts and have been used in the same way. This is not 
surprising since some methods, such as Analytic Hierarchy Process 
(AHP) can contribute to more than one decision making stage. 
Moreover, Buchholz et al. [6] illustrate the stages of a ‘MCA cycle’, 
which is a separate cycle within an ‘adaptive multi-stakeholder 
management cycle’. These stages commonly, but not exclusively 
comprise: (i) C&I weighting or ranking, (ii) assessment and 
ranking of alternatives and (iii) sensitivity analysis. 

Against this background, this paper presents a methodology to 
select suitable MCA methods covering the stages of the ‘MCA cycle’ 
for decision making in the renewable energy sector, which is 


challenging due to the mentioned complexity of participatory 
sustainable decision making in the energy sector, as well as due 
to the large number and complexity of MCA methods. 


2. Methodology for MCA method selection 


The methodology presented in this paper builds on literature 
(e.g. [7,16]). For instance, Guitouni and Martel [15] provide a 
framework to develop a selection methodology and tentative 
guidelines to support selecting an appropriate MCDA method. 
Furthermore, Buchholz et al. [6] analyzed the applicability of four 
MCA tools for bioenergy systems assessments by scrutinizing their 
structural approach using a set of nine criteria to evaluate 
methodological implications, technical aspects and the application 
value of MCA methods. 

Following the methodology recommendations from the 
reviewed literature, the methodology presented in this paper 
consists of both, generic and case-study specific parts. First, MCA 
methods and criteria to select MCA methods from literature were 
identified and reviewed. Then, the methodology was applied in a 
case study involving the main steps: development of a generic 
decision making situation, development of a set of appropriate 
criteria for MCA method selection, pre-selection of MCA methods 
and scoring of the pre-selected MCA methods against the devel- 
oped selection criteria. In the subsequent sections these processes 
are described and discussed in detail. 


2.1. Identification of weighting and MCA methods applied in the 
energy and renewable energy sectors 


MCA methods have been extensively addressed in academic 
articles covering the energy sector in general, as well as the 
renewable energy sector. Table 1 summarizes reviewed literature 
on weighting and MCA approaches applied in these sectors. The 
table illustrates only a certain fraction of studies in these sectors. 
However, in contrast to similar summaries (e.g. [4]) it focuses on 
more recently applied methods, as well as stakeholder involve- 
ment. For a detailed review of methods in the different stages of 
multi-criteria decision making for sustainable energy we recom- 
mend Wang et al. [14]. Further rather theoretical studies in this 
field include for instance Chatzimouratidis and Pilavachi [17], 
Pilavachi et al. [18], Chatzimouratidis and Pilavachi [19]. 

It can be concluded that the vast number of applied MCA 
methods to support sustainable decision making in the energy 
sector in general, as well as the renewable energy sector makes it 
imperative to take a systematic approach to select MCA methods 
for individual projects or case studies. Further notable is that 
frequently several different suitable weighting and/or MCA meth- 
ods are applied to a decision making problem to look at problems 
from various angles. This can help cross-checking and verifying the 
validity of MCA method results before arriving at the final ranking 
of alternatives. 


2.2. Criteria for selecting MCA methods 


Criteria for selecting MCA methods from literature against 
which MCA methods can be scored are described and summarized 
(Table 2) in this section. 

Guitouni and Martel [15] provide general tentative guidelines 
to help choosing an appropriate MCA (specifically MCDA) method. 
Their first guideline is to determine, whether a decision is made by 
one or many decision makers. Second, the preference elucidation 
and modeling has to be considered. This guideline is divided into: 
(a) the preference elucidation mode (e.g. pairwise comparison), 
(b) the timing of preference elucidation, (c) the global decision 


T. Kurka, D. Blackwood / Renewable and Sustainable Energy Reviews 27 (2013) 225-233 227 


Table 1 


Summary of weighting and MCA methods applied in the energy and renewable sectors. 


Scope Weighting and/or MCA method Special features Stakeholder involvement Source 

Sustainability Not specified MCDA method Least cost optimization models for scenarios. | Decision conferencing for C&I selection [20] 
framework to Life cycle impact assessment (LCIA) for C&I (different stakeholders). Experts for C&I 
assess bioenergy performance assessments. Numeral Unit Spread performance assessments 
systems Assessment Pedigree (NUSAP) approach to 

classify qualitative and quantitative 
uncertainties. 

Sustainability MCA based on Analysis and Synthesis Comparison of MCA to single criterion analysis. No [8] 
assessment to of Index at Information Deficiency Stochastic modeling of uncertainty, as well as 
select a country's (ASPID) methodology and a General normalization of indexes by using Non-numeric 
energy system Sustainability Index (ordinal), Non-exact (interval) and Non- 

complete information (NNN-information) with 
respective reliability and probability 

Renewable energy SIMOS and Preference ranking Resource-intensive combination of scenarios Scenario development and improvement of [4] 
scenarios and organization method for enrichment and PMCA. Narrative storylines for scenarios. developed C&I (energy stakeholders and 
participatory evaluation (PROMETHEE) II C&I performance assessments based on own experts). Experts and stakeholders for C&I 
multi-criteria modeling, existing databases and expert performance assessments and weighting. 
analysis (PMCA) assessments. ‘Silent negotiation’ technique for 

C&I ranking 

MCA for bioenergy Analytic Hierarchy Process (AHP), Criteria for MCA method selection. Sensitivity Stakeholders and experts involved in C&I [6] 

systems DELTA , PROMETHEE II, Novel analyses on C&I weighting and performance weighting and performance assessments 
Approach to imprecise Assessment assessment results 
and Decision Environments (NAIADE) 

Appraisal of SIMOS and PROMETHEE (Decision Lab) Stakeholder involvement in scenario and C&I [21] 
national energy development, individual C&I weighting and 
scenarios — discussion of MCA results. Experts involved in 
renewable energy C&I performance assessments 
use 

Sustainable MCA based on fuzzy sets synthesis No [22] 
development ofa technique 
city's energy 
system 

Alternative power PROMETHEE Comparison of MCDA and Cost-Benefit-Analysis Official authorities involved in scenario [23] 
generation (CBA) development 
scenarios 

Small-scale energy MACBETH (Measuring Attractiveness Performance assessments involving LCA and Households, energy professionals and local [9] 
technology by a Categorical Based Evaluation CBA (including calculation of Net Present council members involved in C&I performance 
applications in TecHnique) Values) assessments and weighting 
local 
governments 

Optimization of Elimination et choice translating Sensitivity analysis in relation to the preference, Local authorities and private actors involved in [24] 
decentralized reality (ELECTRE) III indifference, veto thresholds and importance C&I weighting and sensitivity analysis 
energy systems coefficients 

Formulation of MCDA with linguistic variables use Linguistic decision support system (L-DSS) Working groups with national energy actors [25] 
sustainable (fuzzy set theory) incorporating linguistic ordered weighted involved in C&I and alternatives selection, C&I 
technological averaging (LOWA), the 2-tuple LOWA and the weighting and performance assessments 
energy priorities OWMAX (ordered weighted minimum 

operators) to deal with imprecision 

Public and Simple additive approach Policy stakeholders and informed local [26] 
stakeholder members of the public involved in scenario 
perceptions of development, C&I weighting and performance 
bioenergy assessments, as well as appraisal of final 
scenarios weighted scores 

Energy policy and Integrated fuzzy VIKOR Integrated methodology to select energy Experts involved in C&I weighting [27] 
production site (VISekriterijumsko KOmpromisno alternatives and sites 
planning for a city Rangiranje) and AHP methodolgy 

Land suitability AHP Geographic Information System (GIS) modeling No [28] 
assessment for to locate optimal sites. Performance 
spatially assessments for distance-related C&I combined 
distributed with weights obtained from AHP application 
anaerobic 
digester 

Evaluation of Analytic Network Process (ANP) LCA use Experts involved in C&I weighting [29] 


alternative fuels 
for electricity 
generation 


(Super Decisions) 


making preference structure obtained using the aggregation 
procedure and resulting from that (d) the specific alternative 
order type. Thirdly, the decision makers' desired outcomes have 
to be taken into account. An example for such an outcome can be a 
ranking of alternatives. The fourth guideline is about a MCA 
method's ability to process available input data properly. Required 


data should easily be provided by stakeholders. Where qualitative 
data is used, it can make a significant difference to the quality of 
decision making, if a MCA method is capable to incorporate that 
data. Another guideline is about dealing with the compensation 
degree of a MCA method. However, Guitouni and Martel [15] 
admit that this aspect is difficult to evaluate. They also advise to 
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Table 2 
Criteria for MCA method selection from literature. 


Selection criteria Source 
Multi-stakeholder inclusion 6,7,15] 
Determining of C&I weights 7,15;16] 
Handling of qualitative and quantitative data 6,7,15,16] 
Shape and form of final results 15,16] 
Compensation degree 15] 
Meeting the method's fundamental hypothesis 15] 
Support by a software package 15] 
Measures to deal with uncertainty 6,7] 
Country of application 7 
Type of data used 7 
Nature and context of the problem 7 
Spatial scale 7 
Type and selection of stakeholders 7,13] 
Inclusion of C&I hierarchies 6 

Use of thresholds 6 

Ease of computation 6,13] 
Dynamic re-evaluation 6 
Transparency and likely user acceptance of results 6,13] 
Communication of decision process and results 6,13] 
Core process 16] 
Necessity to quantify the relative importance of C&I 16] 
Number and type of outranking relations 16] 
Control of consistency 16] 


bear in mind how MCA methods meet their fundamental hypoth- 
eses. However, the information this criterion provides can be 
regarded as being more descriptive and less suitable for MCA 
method comparison and selection. The final guideline they men- 
tion is about the availability of software packages to apply a MCA 
method. The authors raise concerns that in many cases decision 
aid package software is applied without deeper knowledge of the 
procedure. On the other hand, they argue that it can occur that a 
MCA method, which would be suitable for a decision making 
situation is not employed simply because a software package is not 
user-friendly and/or does not allow integration of a variety of MCA 
methods. 

Mendoza and Martins [7] critically reviewed and systematized 
applied MCA (specifically MCDM) methods in natural resource 
management using a number of criteria and characteristics. The 
most relevant of those indicate, (a) if a MCA method deals 
explicitly with uncertainty, (b) if more than one decision-maker 
is involved and (c) the moment when modeling of C&I preferences 
through stakeholder's involvement takes place. In regard to the 
last aspect, Mendoza and Martins [7] reviewed academic papers 
using three different categories: (a) ‘a priori’, (b) interactive and 
(c) posterior articulations. ‘A priori’ means that the stakeholder 
involvement starts with the modeling of preferences. On the other 
hand, ‘interactive articulation’ of preferences results in the most 
preferred solution or alternative through stakeholders answering 
specific questions. For ‘posterior articulation’ of preferences ‘non- 
dominated’ solutions are generated prior to the evaluations made 
by the stakeholders. When elucidation methods are carried out in 
group decision making situations the individual stakeholder pre- 
ferences can be synthesized in various ways, whereas Mendoza 
and Martins [7] mention that aggregating has been established as 
a main aspect in methodological adaptations of MCA to group 
decision making. Furthermore, they point out that many aspects in 
the field of natural resources management cannot be estimated or 
predicted sufficiently, which contributes to uncertainty in decision 
making. However, if uncertainty is not addressed sufficiently, 
suboptimal or even inappropriate alternatives are potentially 
selected and implemented ultimately resulting in unsustainable 
outcomes. Other selection criteria mentioned [7] (e.g. ‘country of 
application’, ‘spatial scale’, ‘nature and context of problems’ or 
‘type of data used’) provide more descriptive information and, 
therefore, are less relevant for the selection of MCA methods. 


The mentioned criterion ‘type and selection of stakeholders’ can 
also be regarded as being irrelevant for MCA method selection, 
this process usually takes place as a separate step earlier in a 
decision making process (e.g. [2,30,31]). 

Buchholz et al. [6] analyzed how applicable MCA methods for 
bioenergy systems assessments are by scrutinizing their structural 
approach using nine selection criteria. The first selection criterion 
covers ‘stakeholder inclusion in decision making’ and takes 
account of the fact that renewable energy developments, such as 
bioenergy developments, typically involve a relatively high num- 
ber of stakeholders. In this connection, it is pointed out that many 
of past projects were unsuccessful when stakeholders were not 
engaged appropriately. The second selection criterion covers the 
‘application of qualitative data’. Technical aspects are taken into 
account by three criteria. ‘Measures to deal with uncertainty’ is the 
first among those and is justified by the significance and varied 
forms of uncertainty, which can be dealt with by MCA methods at 
the different stages of a decision making process. Another techni- 
cal aspect is the ‘inclusion of criteria hierarchies’. Hierarchies assist 
in reducing complexities and allow facilitated communication of 
decision making results and strategies to stakeholders without 
technical knowledge. The third technical criterion ‘use of thresh- 
olds’ is about implications of applied numerically precise thresh- 
olds, which are helpful in dealing with data uncertainty. Four 
criteria cover the applicability of MCA methods. ‘Ease of computa- 
tion’ addresses the technical user-friendliness and flexibility of 
MCA methods. In this respect, the criterion also takes into account 
the differences among MCA methods regarding problem structur- 
ing, parameter setting and sensitivity analysis. Related to this 
criterion is ‘dynamic re-evaluation’, which is about a MCA method's 
ability to allow re-evaluating decisions based on new structures, 
insights or data. This ability supports flexible and adaptive learn- 
ing cycles as proposed by Gamboa and Munda [32]. An important 
aspect when involving numerous stakeholders is the application of 
simple math and intuitive methods to improve the acceptance of 
MCA methods results. The criterion ‘transparency’ covers this 
aspect. Related to this criterion is the last criterion ‘communica- 
tion of decision process and results’. Decent abilities of MCA 
methods in respect to this aspect can be critical when numerous 
stakeholders are involved. Apart from aspects related to stake- 
holder involvement and easy handling, considering these aspects 
covering acceptance and understanding are also recommended by 
Huang et al. [13] under certain circumstances. 

Ozcan et al. [16] describe a comparative analysis of MCA 
(specifically MCDM) methodologies and implement a warehouse 
location selection problem. Although, not specifically focused on 
sustainability or energy decision making, their analysis and 
comparison of MCA methods including AHP, TOPSIS, ELECTRE 
I-III provide useful inputs for the process of selecting an appro- 
priate MCA method. The several methods reviewed in the paper 
are differentiated by seven characteristics or criteria: (a) core 
process, (b) necessity to quantify the relative importance of C&l, 
(c) determining of C&I weights, (d) number and type of outranking 
relations, (e) consistency check, (f) problem structure and (g) final 
results. The first characteristic provides rather descriptive infor- 
mation and can be regarded as being less suitable for MCA method 
selection. The criteria about quantifying the relative importance of 
C&I and determining C&I weights are both focusing on the ‘MCA 
cycle’ step ‘C&I weighting or ranking’. The first criterion does not 
serve to discriminate sufficiently between methods to support the 
selection process, because almost all MCA methods require the 
relative importance of C&I to be quantified. On the other hand, 
the second criterion can be considered as being important, 
because C&I weighting is an essential part of MCA methods and 
differences between methods in this regard are significant. Not 
every MCA method has a specific weighting method incorporated 
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and for some C&I weights are provided without any calculations. 
However, for the latter methods commonly linear and vector 
normalizations are used [33]. The described criterion ‘number 
and type of outranking relations’ can be regarded as being too 
specific and focused on outranking methods for selecting MCA 
methods. On the other hand, it can be regarded as being important 
to measure and control the consistency of stakeholders' prefer- 
ences. The criterion ‘consistency check’ shows, if MCA methods 
allow controlling consistency. The criterion ‘problem structure’ 
covers numbers of C&I and alternatives, which have to be 
compared, as well as a method's ability to handle quantitative 
and/or qualitative data. This aspect is only relevant for decision 
making problems with large numbers of C&I and alternatives, and 
as mentioned previously, only if qualitative data is used. The 
criterion ‘final results’ covers the shape in which acquired MCA 
results are available. This criterion can be regarded as a non- 
essential feature for selecting suitable MCA methods. 


3. Application of the methodology for MCA method selection 


This section illustrates how the methodology for MCA method 
selection was applied. First, the case study is described. Second, 
the most appropriate set of MCA method selection criteria was 
developed based on the reviewed literature (Section 2.2). Then, 
some MCA methods were pre-selected and compared against 
these criteria to choose the most appropriate MCA method for 
the case study. 


3.1. Case study description 


For the application of the methodology to select a MCA 
method, basically a generic decision making situation was 
assumed, in which renewable energy developments are planned. 
In terms of the MCA method selection itself, it was further 
assumed that MCA methods cover the following steps of the 
MCA cycle: (i) C&I weighting or ranking, (ii) assessment and 
ranking of alternatives and (iii) sensitivity analysis. Also, it was 
assumed that stakeholders are involved in a single event for C&I 
weights determination. In terms of C&I performance assessments, 
it was assumed that the performances are assessed objectively and 
that stakeholders agree exclusively on employing quantitative C&l. 
Choosing this generic decision making situation was intended to 
demonstrate the broad application potential of the presented 
approach to select MCA methods in the renewable energy sector. 


3.2. Choice of criteria for MCA method selection 


Based on the literature review (Section 2.2), a set of most 
appropriate MCA selection criteria for the case study was devel- 
oped. Table 3 illustrates these criteria, which are also briefly 
described for clarification. 

The illustrated selection criteria were chosen by considering 
their relevance for the case study, but also by bearing in mind how 


Table 3 
Criteria for MCA method selection for the case study. 
Selection criteria Description: 
Measures to deal with uncertainty 
User-friendliness and flexibility 


re-evaluations? 
Transparency and communication 


independent they are from each other. For instance, criteria related 
to ‘transparency’, ‘inclusion of C&I hierarchies’ and ‘communica- 
tion of decision process and results’ can be employed as separate 
criteria. However, when looking at the descriptions of criteria [6], 
relatively strong interrelationships can be identified. Good com- 
munication leading to easy comprehension is supported by trans- 
parency, but on the other hand good communication assists in 
increasing acceptance and transparency. C&I hierarchies help to 
reduce complexity and as a consequence facilitate communication 
or processes and results. This again improves stakeholder com- 
prehension. Therefore, for the case study these three criteria were 
merged into a single criterion called ‘transparency and commu- 
nication’. In general, the selection criteria were kept as independent 
as possible for the case study. However, some interrelationships 
exist. For instance the criterion ‘multi-stakeholder inclusion’ is 
related to the criteria ‘user friendliness and flexibility’, as well as 
‘transparency and communication’ to a certain extent. Also, it has 
to be highlighted that a selection criterion comparing, whether 
MCA methods are able to process qualitative C&I data was not 
required, because as mentioned (Section 3.1) only quantitative C&I 
were considered. Also, employing exclusively quantitative C&I, 
which can be assessed objectively, affected the scoring of the 
MCA methods against the selection criterion ‘measures to deal 
with uncertainty’, as uncertainty related to determining C&I 
weights could be focused on. This demonstrates that the processes 
of choosing appropriate selection criteria and the selection of MCA 
methods itself depend heavily on a project’s or case study's 
specific circumstances. In this connection, particularly, the way 
multiple stakeholders were involved plays a major role. For 
instance, if stakeholders want to be involved in a single event 
for C&I weights determination - as it was assumed in case study's 
decision making situation (Section 3.1) - C&I weighting has to be 
undertaken ‘a priori’. Therefore, ‘a priori’ C&I weighting had to be 
considered as a prerequisite in selecting MCA methods in the case 
study. Similarly, in such a decision making situation in which 
stakeholders agree exclusively on quantitative C&I with relatively 
certain performance values, capabilities of MCA methods to deal 
with qualitative C&I and uncertainty are not required or only 
relevant to a certain degree, respectively. 


3.3. Pre-selection of MCA methods 


In this section it is described how three MCA methods were 
pre-selected in order to test the methodology in the case study. 
This pre-selection was mainly based on the popularity of MCA 
methods. 

Huang et al. [13] identified and classified over 300 papers on 
environmental applications of MCA (specifically MCDA), which were 
published between 2000 and 2009. The papers were classified by their 
environmental application area, decision or intervention type and 
three main MCA approaches were identified and employed for their 
analysis: ‘Outranking’, ‘AHP/ANP (Analytic Network Process)’ and 
‘MAUT/MAVT’. When looking at papers about MCA methods in the 
application area ‘Energy’ published between 2000 and 2009, AHP/ANP 


To what degree does a MCA method address uncertainty in its different stages? 
How easy is the computation of MCA processes? Is the MCA method user friendly, flexible and adaptive and does it allow 


How simple and intuitive is a MCA method to increase transparency and stakeholder acceptance? Are hierarchies 


incorporated to help reducing complexity and to support communication of processes and results in order to improve 


stakeholder comprehension? 
Multi-stakeholder inclusion 


How easy is it to involve multiple stakeholders in the MCA process? 
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approaches were predominantly employed, followed by PROMETHEE, 
ELECTRE and MAUT/MAVT methods accounting for 42, 12, 9 and 9% 
respectively. Moreover, a trend towards AHP/ANP methods, indicated 
by the growing number of their applications over that decade, was 
identified. AHP/ANP methods were also identified as the most popular 
approaches (48%) when looking at published papers about applica- 
tions of MCA methods with stakeholder participation. 

For the case study, most popular MCA methods according to 
Huang et al. [13] were compared against the selection criteria 
(Table 3). These pre-selected MCA methods were: AHP [34], DELTA 
[35] and PROMETHEE [36] representing MAVT, MAUT and out- 
ranking methods, respectively [37]. In order to compare only one 
method representing a main MCA approach, the popular ELECTRE 
method was excluded from the comparison due to its lower 
popularity compared to PROMETHEE, which is also an outranking 
method. Also, AHP was focused on in this comparison rather than 
ANP, because AHP is significantly more applied in the environ- 
mental management and agriculture application area [38]. 

Notable is that MCA combined fuzzy methodologies, which 
involve a series of mathematical steps and are basically fuzzy 
extensions to conventional MCA methods (e.g. fuzzy outranking 
methods and fuzzy utility theory), have also been applied in 
energy decision making situations various times (e.g. [25,39,40]). 
However, due to several reasons they are unpopular within the 
MCA community and have predominately been discussed in 
academic literature and conducted experimentally [2]. For 
instance, particularly for AHP and ANP the accuracy of the fuzzy 
approach is criticized [41]. Therefore, the practical value of MCA 
combined fuzzy methodologies can be substantially questioned 
and it is even assumed that their future application will be flawed 
in regard to better decision making [38]. Consequently, these 
approaches were excluded from the process of selecting MCA 
methods in the case study. 


3.4. Scoring of pre-selected MCA methods 


Each of the three pre-selected MCA methods ‘AHP’, ‘DELTA’ and 
‘PROMETHEE’ was compared against the four selection criteria 
described previously (Table 3). In this process, the MCA methods 
were assessed using a three-point scale with ‘low’, ‘medium’ and 
‘high’ scores. Additionally, specific freely downloadable or com- 
mercially available software packages were considered to compare 
them against some of the selection criteria: 


e ‘Super Decisions’ (AHP) 
© ‘DecidelT’ (DELTA method) 
è ‘Decision Lab’ (PROMETHEE II) 


The software packages were selected due to their common 
applications in related fields such as renewable energy, natural 
resource Management or participatory assessments. Based on a 
literature review and by considering the case study’s circum- 
stances (Section 3.1), the software packages' scoring against some 
of the selection criteria contributed to identifying the most 
suitable MCA method and its software package for the case study. 
The following two examples illustrate how the scoring against two 
selection criteria was undertaken. 


3.4.1. User friendliness and flexibility 

‘User-friendliness and flexibility’ was one of the four criteria 
against which the AHP, DELTA and the PROMETHEE II methods 
were compared. The scores for this criterion reflect several aspects 
including ease of computation and technical user-friendliness of 
the reviewed methods and their software packages, as well as 


their flexibility and adaptability in terms of allowing re-evaluation 
of decisions with new data inputs or structures. 

‘Super Decisions’ scored ‘medium’ against this criterion. It 
allows easy computation with its straightforward structure and 
its simple framework to build and assess scenarios and alterna- 
tives. Particularly, data input of the pairwise comparison data is 
perceived as simple and convenient by users [2]. Furthermore, 
data input requirements are minimal and free downloadable 
software package versions of ‘Super Decisions’ are available. 
Looking at these aspects in isolation the user friendliness of ‘Super 
Decisions’ is high. However, it is relatively difficult to apply 
sensitivity analyses and dynamic re-evaluations. Performing sen- 
sitivity analyses is difficult, because the software package's cap- 
ability for analysis is restricted to one criterion at a time. In terms 
of dynamic re-evaluations, direct comparisons between existing 
assessments and assessments with new data input are not 
possible. Comparing those assessments is only feasible indirectly 
through re-structuring data and running separate assessments 
[6,15]. Also, the ‘rank reversal’ phenomenon [42] has to be 
considered when decisions want to be re-evaluated. However, this 
phenomenon has only to be taken account of, in cases where C&l 
performance values are included in the assessments. 

In terms of ease of computation and technical user-friendliness, 
the capabilities of the commercially available DecidelT software 
package can be considered as decent with its advanced user 
interface and intuitive logic to structure and define C&I, as well 
as conducting sensitivity analysis. Furthermore, it requires low 
minimum data input. However, if minimization is the goal of the 
decision making situation, this software package of the DELTA 
method requires manual inversion of data. For example, the goal 
air pollution reduction has to be converted to air quality improve- 
ment. This means an additional step, which reduces DecidelT's 
user friendliness. As for ‘Super Decisions’, re-evaluating decisions 
is difficult as new data input cannot be integrated, which disallows 
direct comparisons between old and new assessments [6]. There- 
fore, the case study's score for ‘user friendliness and flexibility’ was 
considered to be ‘medium’. 

The available software package ‘Decision Lab’ scored ‘high’ 
against ‘user friendliness and flexibility’. It allows relatively easy 
computation in all its application stages. Technical user friendli- 
ness is supported by an easy-to-use interface to structure the 
decision making problem and to input data. This software package 
of PROMETHEE II also offers a user-friendly sensitivity analysis 
with the possibility to have numerous perspectives on the pro- 
blem. This includes the use of ‘walking weights’ and side-to-side 
assessments of scenarios and alternatives [6,15]. In terms of 
dynamic re-evaluation, ‘Decision Lab’ is equally well capable, 
which allows engaging in flexible and adaptive learning cycles. It 
is the only reviewed software package, which offers direct com- 
parisons between existing and new assessments of scenarios and 
alternatives through its scenarios and alternatives analysis, which 
also allows structures to be changed for these comparisons [6]. 


3.4.2. Multi-stakeholder inclusion 

Another criterion against which the three MCA methods AHP, 
DELTA and PROMETHEE II were compared against was ‘multi- 
stakeholder inclusion’. This criterion is about an approach's cap- 
ability to involve multiple stakeholders in the MCA process easily. 
Naturally, the two criteria ‘user friendliness and flexibility’, as well 
as ‘transparency and communication’ are connected to this criterion 
to a certain extent. Thus, the scores of the methods against those 
criteria also contain and cover elements of the approaches' 
abilities to include multiple stakeholders. In line with the scenario 
description (Section 3.1), the three MCA methods AHP, DELTA and 
PROMETHEE II and their software packages were evaluated 
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regarding their coverage of some or all of the mentioned steps of 
the MCA cycle. In other words a holistic perspective looking at all 
those stages was necessary to determine an approach's capability 
for multi-stakeholder involvement. Also in line with the scenario 
description (Section 3.1), it has to be taken into account that it was 
assumed that stakeholders express their opinions towards C&l 
priorities ‘a priori’. All three reviewed MCA methods allow the 
incorporation of C&I weighting during that stage of the decision 
making process. 

The AHP method scored ‘high’ against the selection criterion 
when keeping the case study in mind. One of the core elements of 
AHP is the conversion of subjective assessments of relative 
importance to overall weights [2,6]. The method is the only of 
the three reviewed MCA methods, which works with a specific 
procedure to determine C&I weights - pairwise comparisons 
between C&I - and through that provides ready-to-use C&I 
weights or rankings. Pairwise comparisons significantly reduce 
the likelihood of stakeholders making mistakes and rough estima- 
tions by deriving ratio scale priorities, while eliciting utility 
functions is not required [17]. This allows stakeholders to be easily 
involved. The method also allows group decision making proce- 
dures in assessing and ranking alternatives, as well as in the 
sensitivity analysis stage [6,7]. 

The DELTA method scored ‘medium’ against the criterion 
‘multi-stakeholder inclusion’. This score is justified, because the 
method requires external aggregation of C&I weights through 
pairwise comparison. This makes stakeholder involvement more 
difficult. However, the method allows stakeholder involvement in 
scenarios and alternatives assessment and sensitivity analysis, 
when uncertainty of C&I weights is taken account of [6]. 

The PROMOTHEE II approach also scored ‘medium’ against the 
criterion ‘multi-stakeholder inclusion’. Like the DELTA method it 
relies on externally derived C&I weights or rankings. However, 
stakeholders can be involved in the assessment of scenarios and 
alternatives, as well as in the sensitivity analysis. Furthermore, it 
allows analyzing group decisions and comparing individual opi- 
nions with each other by assigning scenarios to each individual 
stakeholder’s opinion [6]. 


4. Results and discussion 


The scores of the three different MCA methods compared 
against the selection criteria are summarized below (Table 4): 

In the case study, the AHP method scored best with three ‘high’ 
scores and one ‘medium’ score, followed by the DELTA method 
with two ‘high’ and two ‘medium’ scores. PROMETHEE II only 
scored ‘high’ once, ‘medium’ twice and ‘low’ against one criterion. 
Interestingly, this MCA method scored best against the criterion 
‘user friendliness and flexibility’, against which the other methods 
only achieved a ‘medium’ score. Conversely, PROMETHEE II scored 
worst against ‘measures to deal with uncertainty’, whereby the 
results of the other two methods are particularly well against this 
criterion. Further, it can be observed that the scores for the AHP 
and Delta methods against the selection criteria only differ in 


Table 4 
Scoring results of reviewed MCA methods for the case study. 


Selection criterion MCA method 

AHP DELTA PROMETHEE II 
Measures to deal with uncertainty High High Low 
User friendliness and flexibility Medium Medium High 
Transparency and communication High High Medium 
Multi-stakeholder inclusion High Medium Medium 


regard to ‘multi-stakeholder inclusion’, which is not surprising 
when bearing in mind how closely MAVT and MAUT are related 
[1]. The difference in scores is mainly due to the fact that AHP is 
the only of the three methods, which works with a specific 
participatory procedure to determine C&I weights — pairwise 
comparisons — instead of relying on external aggregation of C&l 
weights. 

When reflecting and concluding on the scoring process, it has 
to be addressed that several factors and assumptions influenced 
the results significantly. First, it has to be considered against what 
the MCA methods were compared, i.e. the selection criteria. 
Choosing them was to a certain extent subjective and their choice 
influenced the results of the MCA method selection substantially. 
In other words, using different criteria against which the methods 
were compared, could have led to diverging results. The same 
applies to what was scrutinized against the criteria, i.e. the three 
MCA methods. Although, their pre-selection was justified by their 
popularity and common use in the energy sector, the focus on 
specific software packages representing those methods when 
compared against some of the selection criteria, was rather 
subjective. However, in either instance the subjectivity stemmed 
from the case study’s decision making situation, which had to be 
considered (Section 3.1). Huang et al. [13] confirm this by high- 
lighting the importance of considering the complexity of the MCA 
method selection in respect to influencing scientific, social and 
technical factors, but also in regard to an understanding of the 
process requirements, as well as the degree of available knowledge 
about the specific decision making problem. For the case study the 
most influencing factors in this respect were the fact that the focus 
was on C&I weights excluding C&I performance values, as well as 
quantifiable C&I. In case the stakeholders would have chosen 
predominantly qualitative C&I, another MCA method selection 
process resulting in different outcomes would have been required. 
Features of MCA methods in regard to qualitative C&I, as well as 
the incorporation of C&I performance values in the decision 
making process was therefore not important to the different stages 
of the MCA method selection process in the case study. This again 
demonstrates that the process of choosing appropriate MCA 
methods heavily depends on a project's or case study's specific 
circumstances, which is also confirmed by Guitouni and Martel 
[15] who state a lack of a single ideal and appropriate MCA method 
for all decision making situations. Consequently, it could be argued 
that the complex process of selecting the most suitable MCA 
method could paradoxically call for applying a MCA method to 
help choosing. This however would result in a ‘chicken-or-egg 
situation’, because to employ a MCA method to help choosing a 
MCA method would imply selecting a MCA method in the first 
place, which would still leave one wondering how to select that 
method. Further, it can be observed that the selection criteria 
deemed to be relevant (Section 2.2) were predominantly criteria 
with emphasis on acceptance, understanding and interaction with 
the MCA method. Compared against them AHP is a strong MCA 
method. Buchholz et al. [6] and Chatzimouratidis and Pilavachi 
[17] explain that the method’s general popularity and wide use is 
mainly due to its logical approach and simplicity facilitating these 
aspects, whereas Huang et al. [13] regard available software 
packages, expertise and particularly active user groups as possible 
reasons for this popularity. In this connection it is notable that 
AHP lacks qualities in regard to addressing dependencies and 
interrelations between C&I. Conversely, its extension ANP has 
distinctive qualities in this regard, which can lead to substantial 
improvements in decision making. Due to this reason it is 
expected that ANP will gain more popularity compared to AHP 
in the future [38]. However, due to ANP's higher complexity, ‘user 
friendliness and flexibility’, ‘transparency and communication’ as 
well as ‘multi-stakeholder inclusion’ can be regarded as being 
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reduced compared to AHP. As mentioned (Section 2.2), Huang 
et al. [13] advise MCA users to select methods based on aspects 
related to stakeholder involvement, acceptance, easy handling and 
understanding when MCA methods results can be expected to be 
robust. Such a robustness of MCA methods is generally the case. 
When applying different MCA methods to the same decision 
making problem, it was observed that although the ranking of 
alternatives may change, generally the same alternatives come out 
at the top regardless of the MCA method employed [13]. This is 
due to the mathematical phenomenon that the best ranking 
alternative(s) is/are often so much better than the remaining 
alternatives so that the differences between MCA methods are 
too minor to significantly change the outcome. The reason for this 
is that those differences are most probably affecting all alterna- 
tives equally within C&I assessment levels unless all MCA methods 
would be biased towards a single alternative, which is highly 
unlikely [43,44]. The phenomenon was particularly observed for 
decision making situations with a small number of alternatives 
{13]. Therefore, for these cases distinctive qualities of MCA 
methods would be less significant than for instance selecting most 
appropriate C&I or involving appropriate stakeholders. Neverthe- 
less, regardless the number of alternatives, selecting a suitable 
MCA method in the described systematic manner is an effective 
and valid approach for decision making situations, because it 
illustrates and justifies why a particular MCA method makes more 
sense than other compared methods when specific aspects are 
relevant and focused on. Furthermore, when applying a MCA 
method, which was selected by employing the presented metho- 
dology, the identified limitations and disadvantages of the method 
with respect to the selection criteria can be considered more 
easily. This can facilitate processes in similar decision making 
situations, particularly in the renewable energy sector. In this 
regard, the presented methodology is advantageous compared to 
other approaches to select MCA methods (e.g. [6]), due to its 
generic nature. Although, similar MCA selection criteria are pre- 
sented in related works [6,7,15,16], a methodology to system- 
atically select MCA methods by developing an appropriate set of 
selection criteria, pre-selecting MCA methods and scoring the pre- 
selected methods against the developed set of selection criteria 
has, to our knowledge, not been published yet. 


5. Conclusions 


This paper provides a generic approach to select a suitable MCA 
method covering the stages of the ‘MCA cycle’ for decision making 
in the renewable energy sector. Selecting a suitable MCA method 
for an individual project or case study is challenging due to the 
complexity of participatory sustainable decision making in this 
sector, as well as due to the complexity and large quantity of 
existing MCA methods. 

The provided methodology for selecting MCA methods consists 
of both, generic and case-study specific parts. First, it is proposed 
to identify and review MCA methods and criteria to select MCA 
methods from literature. Second, case study specific steps are 
proposed to be applied including: development of a set of appro- 
priate criteria for MCA method selection, pre-selection of MCA 
methods and scoring of the pre-selected MCA methods against the 
developed selection criteria. 

In general, the process of selecting appropriate MCA methods 
depends heavily on a project’s or case study's specific circum- 
stances. However, the case study's decision making situation 
presented in this paper was deliberately developed in a rather 
generic way allowing a wide range of application potential for 
decision making situations within the renewable energy sector. 
The criteria for MCA method selection deemed to be relevant were 


predominantly criteria focusing on acceptance, understanding and 
interaction with the MCA method. AHP scored best when pre- 
selected MCA methods were compared against the selection 
criteria. 

Although, in certain circumstances the results of MCA methods 
do not differ significantly, selecting suitable MCA methods in the 
proposed systematic manner is an effective and valid approach, 
because it helps focusing on specific aspects, which are considered 
important and relevant for decision making processes. Therefore, 
for similar research or decision making situations the proposed 
methodology for MCA selection can be employed, although 
different criteria and/or MCA methods may be chosen for the 
selection process. Furthermore, the identified and reviewed MCA 
methods and selection criteria provide a broad application poten- 
tial for MCA method selection in the energy and renewable energy 
sectors. 
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